Apparatus and method to determine drowsiness of a driver

ABSTRACT

An apparatus and method for determining drowsiness of a driver are provided. The apparatus includes an imaging device that obtains an image of the driver and a controller that determines (e.g., based on the image of the driver) when the driver is in a fatigue state, when the driver is in an eyelid closure state, and when the driver is in a wake-up state. The controller further determines that the driver is in a drowsiness state when after the fatigue state of the driver continues over a predetermined first time interval, the eyelid closure state and the wake-up state of the driver alternately and repetitively occur within a predetermined second time interval.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims the benefit of priority toKorean Patent Application No. 10-2016-0048915, filed on Apr. 21, 2016 inthe Korean Intellectual Property Office, the disclosure of which isincorporated herein in its entirety by reference.

BACKGROUND Field of Invention

The present disclosure relates to an apparatus and a method fordetermining drowsiness of a driver, and more particularly, to anapparatus and a method that obtains an image of a driver and determinethe drowsiness of the driver based on the image of the driver.

Description of Related Art

Currently, vehicle transportation methods include two-wheel motorcycle,a train, a ship, an airplane, and the like which enable people to freelytravel. However, serious injury and property damage of due to adrowsiness driving may occur while a vehicle is being driven.Accordingly, various methods for preventing drowsiness of a driver havebeen researched.

In particular, when a continuous eyelid closure time of a driver isgreater than a first threshold (dTH) an early warning mode is enteredand an early warning is maintained until the continuous eyelid closuretime of the driver is less than 1 dTH which is less than the firstthreshold (dTH) to warn a driver of the drowsiness driving. However,when, a warning is output regarding the drowsiness driving based on thethreshold of the continuous eyelid closure time, the threshold is set tobe high to prevent a misdetection. Accordingly, a non-detection of thedrowsiness driving and a warning delay occur.

[NOTE: Please include the above references in Invention DisclosureStatement.] Further, when an eye image of a driver is obtained, a pupilof the eye that is not viewed is determined to be a closed eye.Accordingly, when the closed eye is detected for a constant thresholdperiod of time, the driver is determined to be in a drowsiness state.However, it may be difficult to obtain a pupil image from an infraredimage obtained by photographing the driver due to a disturbance ofambient light, reflection of glasses, or the like, and determiningdrowsiness based on the continuous eyelid closure time may be difficult.

The above information disclosed in this section is merely forenhancement of understanding of the background of the invention andtherefore it may contain information that does not form the prior artthat is already known in this country to a person of ordinary skill inthe art.

SUMMARY

The present disclosure provides an apparatus and a method fordetermining drowsiness of a driver while a vehicle is being drivenwithin a minimal time interval. Further, an aspect of the presentdisclosure provides an apparatus and a method for determining drowsinessof a driver capable of reducing misrecognition of a drowsiness drivingby a cluster or audio video navigation (AVN) operation.

According to an exemplary embodiment of the present disclosure, anapparatus for determining drowsiness of a driver may include an imagingdevice configured to obtain an image (e.g., a face or other recognitioncharacteristics) of the driver and a controller configured to determinefrom the image of the driver when the driver is in a fatigue state, whenthe driver is in an eyelid closure state, and when the driver is in awake-up state, and configured to determine that the driver is in adrowsiness state when after the fatigue state of the driver continuesover a predetermined first time interval, the eyelid closure state andthe wake-up state of the driver alternately and repetitively occurwithin a predetermined second time interval. The controller may beconfigured to determine when the driver is in the fatigue state based onat least one of when a percentage of an eye closure of the driverexceeds a preset percentage during a predetermined third time intervalor when a face motion of the driver is within a predetermined fourthtime interval.

The apparatus may further include a vehicle driving state obtainerconfigured to obtain a driving state of the vehicle which is driven bythe driver, wherein the controller may be configured to determinewhether the driver is in the fatigue state based on at least one of whenthe vehicle deviates from a driving lane and whether or the vehicle isdriven in a zigzag direction from the driving state of the vehicle. Thecontroller may be configured to determine whether the driver is in thewake-up state based on at least one of whether an eye blink of thedriver occurs within a predetermined fifth time interval or when a facemotion of the driver exceeds a preset degree.

According to another exemplary embodiment of the present disclosure, amethod for determining drowsiness of a driver may include obtaining byan imaging device an image of the driver, determining, by a controller,from the image of the driver when the driver is in a fatigue state, whenthe driver is in an eyelid closure state, and when the driver is in awake-up state and determining, by the controller, that the driver is ina drowsiness state when, after the fatigue state of the driver continuesover a predetermined first time interval, the eyelid closure state andthe wake-up state of the driver occur alternately and repetitivelywithin a predetermined second time interval. The determination of whenthe driver is in the fatigue state may be determined based on at leastone of when an eyelid closure time of the driver exceeds a predeterminedthird time interval or when a face motion of the driver is within apredetermined fourth time interval.

The method may further include detecting a driving state of a vehiclewhich is driven by the driver, wherein when the driver is in the fatiguestate may be determined based on at least one of the vehicle's deviatesfrom a driving lane a lane or when the vehicle is driven in a zigzagdirection from the driving state of the vehicle. The determination ofwhen a driver is in the wake-up state may be determined based on atleast one of when an eye blink of the driver occurs within apredetermined fifth time interval or when a face motion of the driverexceeds a preset degree.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will be more apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings

FIG. 1 is an exemplary block diagram of an apparatus to determinedrowsiness of a driver according to an exemplary embodiment of thepresent disclosure;

FIG. 2 is an exemplary graph illustrating a face motion of a driver todetermine when the driver is in a fatigue state according to anexemplary embodiment of the present disclosure;

FIG. 3 is an exemplary diagram illustrating a pattern for determiningdrowsiness of a driver according to an exemplary embodiment of thepresent disclosure;

FIG. 4 is an exemplary flow chart illustrating a method for determiningdrowsiness of a driver according to an exemplary embodiment of thepresent disclosure; and

FIG. 5 is an exemplary block diagram illustrating a computing systemexecuting the method for determining drowsiness of a driver according toan exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some exemplary embodiments of the present disclosure willbe described in detail with reference to the illustrative drawings. Itis to be noted that in giving reference numerals to components of eachof the accompanying drawings, the same components will be denoted by thesame reference numerals even though they are shown in differentdrawings. Further, in describing exemplary embodiments of the presentdisclosure, well-known constructions or functions will not be describedin detail in the case in which they may unnecessarily obscure theunderstanding of the exemplary embodiments of the present disclosure.

In describing the components of exemplary embodiments of the presentdisclosure, terms such as first, second, A, B, (a), (b), etc. can beused. These terms are used only to differentiate the components fromother components. Therefore, the nature, order, sequence, etc. of thecorresponding components are not limited by these terms. In addition,unless defined otherwise, it is to be understood that all the terms usedin the specification including technical and scientific terms have thesame meaning as those that are understood by those skilled in the art towhich the present disclosure pertains. It should be understood that theterms defined by the dictionary are identical with the meanings withinthe context of the related art, and they should not be ideally orexcessively formally construed unless clearly defined otherwise in thepresent application.

Furthermore, control logic of the present invention may be embodied asnon-transitory computer readable media on a computer readable mediumcontaining executable program instructions executed by a processor,controller/control unit or the like. Examples of the computer readablemediums include, but are not limited to, ROM, RAM, compact disc(CD)-ROMs, magnetic tapes, floppy disks, flash drives, smart cards andoptical data storage devices. The computer readable recording medium canalso be distributed in network coupled computer systems so that thecomputer readable media is stored and executed in a distributed fashion,e.g., by a telematics server or a Controller Area Network (CAN).

Unless specifically stated or obvious from context, as used herein, theterm “about” is understood as within a range of normal tolerance in theart, for example within 2 standard deviations of the mean. “About” canbe understood as within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%,0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise clear fromthe context, all numerical values provided herein are modified by theterm “about.”

It is understood that the term “vehicle” or “vehicular” or other similarterm as used herein is inclusive of motor vehicle in general such aspassenger automobiles including sports utility vehicles (SUV), buses,trucks, various commercial vehicles, watercraft including a variety ofboats, ships, aircraft, and the like and includes hybrid vehicles,electric vehicles, combustion, plug-in hybrid electric vehicles,hydrogen-powered vehicles and other alternative fuel vehicles (e.g.fuels derived from resources other than petroleum).

FIG. 1 is an exemplary block diagram of an apparatus for determiningdrowsiness of a driver according to an exemplary embodiment of thepresent disclosure. FIG. 2 is an exemplary graph illustrating a facemotion of a driver to determine when the driver is in a fatigue stateaccording to an exemplary embodiment of the present disclosure. FIG. 3is an exemplary diagram illustrating a pattern for determiningdrowsiness of a driver according to an exemplary embodiment of thepresent disclosure.

First, referring to FIG. 1, an apparatus for determining drowsiness of adriver may include an imaging device 100 (e.g., a camera, a video cameraor the like), a vehicle driving state obtainer 200, and a controller300. The controller may be configured to operate the imaging device 100and the driving state obtainer 200. However, the components illustratedin FIG. 1 are not essential components. Therefore, the apparatus fordetermining drowsiness of a driver having components more or less thanthe components illustrated in FIG. 1 may also be implemented. Theimaging device 100 may be configured to obtain an image (e.g., a facialrecognition) of the driver. The imaging device 100 may be a cameraconfigured to photograph a face (or a portion thereof) of the driverwithin a vehicle and may also be an image receiver configured to obtainan image from the camera. Further, the vehicle driving state obtainer200 may be configured to obtain a driving state of a vehicle which isdriven by the driver. The vehicle driving state obtainer 200 may obtaininformation including but not limited to a driving direction of thevehicle, driving velocity of the vehicle, a distance between the vehicleand a vehicle in front, whether the vehicle deviates from the drivinglane (e.g., the vehicle violates a lane of a road), and the like.

Further, the controller 300 may be configured to determine a drowsinessstate of the driver, may determine when the driver is in a fatiguestate, when the driver is in an eyelid closure state, and when thedriver is in a wake-up state based on the image of the driver obtainedby the imaging device 100 and the driving state of the vehicle obtainedby the vehicle driving state obtainer 200. Accordingly, the controllermay be configured to determine when the driver is in a drowsiness statebased on when the driver is in the fatigue state, when the driver is inthe eyelid closure state, and when the driver is in the wake-up state.

The controller 300 may consider when a percentage of an eye closure(PERCLOS) of the driver during a predetermined time exceeds a constantpercentage, when the face motion of the driver is within a predeterminedrange over a predetermined time, when the vehicle is driven within thedriving lane, when the vehicle is driven in a zigzag pattern, or thelike, to determine when the driver is in the fatigue state. For example,when the constant percentage is set to about 10%, and a time in which aneye of the driver is closed (e.g., is about 6 seconds or greater during60 seconds), the controller 300 may be configured to determine that thedriver is in the fatigue state.

Additionally, referring to FIG. 2, a section in which the face motion ofthe driver is maintained in the range of about 1 second or greater toabout 5 seconds does not exist up to about 2 seconds. However, the facemotion of the driver may be maintained in the range of about 1 second orgreater to about 5 seconds in a section from about 2 seconds to about 3seconds. In particular, when the face motion of the driver is minimalwith respect to a constant time interval, the controller 300 may beconfigured to determine that the driver is in the fatigue state.Further, when the vehicle is driven continuously changing lanes ordeviating from the driving lane (e.g., without keeping within thedriving lane) or when the vehicle is driven in the zigzag pattern fromthe driving state of the vehicle, the controller 300 may be configuredto determine that the driver is in the fatigue state. A method ofdetermining, by the controller 300, when the driver is in the fatiguestate may be a combination of illustrated methods, and are not limitedto the illustrated methods.

The controller 300 may be configured to determine when the driver is inthe eyelid closure state by recognizing when an eyelid of the driver isclosed based on the image of the driver. Further, the controller 300 maybe configured to determine when an eye blink of the driver is a normaleye blink or when the face motion of the driver exceeds a constantdegree to determine when the driver is in the wake-up state. Forexample, the normal eye blink may include an eye blink when the driveris not in the drowsiness state, and may include the eye blink performedwithin a constant time interval. In particular, when the driver blinkseyes in the drowsiness state, a time interval when the eyes of thedriver are closed in one eye blink is a greater duration than the normaleye blink.

The controller 300 may be configured to determine when the driver is inthe wake-up state based on the occurrence of the normal eye blink.Additionally, the face motion of the driver may include a change in adirection to which the face of the driver is directed or a change in aposition of the face. When the face motion of the driver exceeds aconstant degree, the controller 300 may be configured to determine whenthe driver is in the wake-up state. A method of determining, by thecontroller 300, when the driver is in the wake-up state may be acombination of illustrated methods, and are not limited to theillustrated methods.

In other words after the fatigue state of the driver continues for afirst time interval or more than a first time interval, the eyelidclosure state and the wake-up state of the driver during a third timerange may occur alternately and repetitively within a second time, thecontroller 300 may be configured to determine that the driver is in thedrowsiness state based on when the driver is in the fatigue state, whenthe driver is in the eyelid closure state, and when the driver is in thewake-up state which are determined as described above.

For example, when the first time is set to about 5 seconds, the secondtime is set to about 4 seconds, and the third time range may be set toabout 0.5 seconds or greater to about 1.5 seconds or less will bedescribed as an example with reference to the pattern illustrated inFIG. 3 from which the controller 300 may be configured to determine thedrowsiness state of the driver. A time interval in which the fatiguestate continues may be about 7 seconds, and continues for about 5seconds, which is the first time interval. Thereafter, the eyelidclosure state of the driver may occur during about 1 second, which iswithin the third time interval. Additionally, after the eyelid closurestate occurs, the wake-up state occurs, and after the eyelid closurestate again occurs for about 1 second, the wake-up state occurs. Inother words, the eyelid closure state and the wake-up state may occuralternately and repetitively. In particular, the eyelid closure stateand the wake-up state may occur alternately and repetitively betweenabout 7 seconds and about 10 seconds and occur within about 4 seconds,which is the second time interval. Accordingly, when the fatigue stateoccurs, the eyelid closure state and the wake-up state of the driveroccur, and the controller 300 may be configured to determine that thedriver is in the drowsiness state.

Hereinafter, a method for judging drowsiness of a driver will bedescribed in detail with reference to FIG. 4 based on the configurationsdescribed above. FIG. 4 is an exemplary flow chart illustrating a methodfor determining drowsiness of a driver according to an exemplaryembodiment of the present disclosure. Referring to FIG. 4, a method fordetermining drowsiness of a driver according to an exemplary embodimentof the present disclosure may include obtaining an image of a driverS100, determining from the image of the driver when the driver is in afatigue state, when the driver is in an eyelid closure state, and whenthe driver is in a wake-up state S200 and determining that the driver isin a drowsiness state, when, after the fatigue state of the drivercontinues over a predetermined first time interval, the eyelid closurestate and the wake-up state of the driver alternately and repetitivelyoccur within a predetermined second time interval 5300.

The imaging device 100 may be configured to obtain the image (e.g. faceimage) of the driver S100. The imaging device 100 may be configured toobtain the image of the driver by photographing a face of the driver(e.g., or a portion thereof) and may also obtain an image from a cameraas an image receiver. Further, the vehicle driving state obtainer 200according to an exemplary embodiment of the present disclosure may beconfigured to obtain a driving state of a vehicle which is driven by thedriver S200. The vehicle driving state obtainer 200 may obtaininformation that includes a driving direction of the vehicle, drivingvelocity of the vehicle, a distance between the vehicle and a vehicle infront, whether or not the vehicle violates a lane of a road, and thelike. The various driving state information may be detected usingvarious sensors mounted within the vehicles.

The controller 300 may be configured to determine from the image of thedriver when the driver is in the fatigue state S200, when the driver isin the eyelid closure state, and when the driver is in the wake-upstate. The controller 300 may use information related to when apercentage of an eye closure (PERCLOS) of the driver during apredetermine time exceeds a constant percentage, when a face motion ofthe driver is within a predetermined range over a predetermine time,when the vehicle keeps the lane, whether the vehicle is driven in azigzag pattern, or the like, to determine when the driver is in thefatigue state. Additionally, the controller 300 may be configured todetermine when the driver is in the eyelid closure state by recognizingwhen an eyelid of the driver is closed based on the image of the driver.In particular, the controller 300 may consider when an eye blink of thedriver is a normal eye blink or when the face motion of the driverexceeds a constant degree to determine when the driver is in the wake-upstate.

After the fatigue state of the driver continues for a first timeinterval or greater, the eyelid closure state and the wake-up state ofthe driver during a third time range occur alternately and repetitivelywithin a second time 5300. The controller 300 may be configured todetermine that the driver is in the drowsiness state based on thedriver's fatigue state, when the driver is in the eyelid closure state,and when the driver is in the wake-up state which are determined asdescribed above. The controller 300 may further be configured todetermine the drowsiness state based on the pattern illustrated in FIG.3, and since a detailed method thereof has been described above withreference to FIG. 3, it will be omitted.

FIG. 5 is an exemplary block diagram illustrating a computing systemexecuting the method for determining drowsiness of a driver according toan exemplary embodiment of the present disclosure. Referring to FIG. 5,a computing system 1000 may include at least one processor 1100, amemory 1300, a user interface input device 1400, a user interface outputdevice 1500, a storage 1600, and a network interface 1700 connected toeach other via a bus 1200. The processor 1100 may be a centralprocessing unit (CPU) or a semiconductor device configured to executeprocesses for instructions which are stored in the memory 1300 and/orthe storage 1600. The memory 1300 and the storage 1600 may includevarious types of volatile or non-volatile storing media. For example,the memory 1300 may include a read only memory (ROM) and a random accessmemory (RAM).

Accordingly, steps in the method or algorithm which is described incontext with the exemplary embodiments disclosed in the presentspecification may be directly implemented in hardware, a softwaremodule, or a combination thereof which is executed by the processor1100. The software module may be resided on a storing medium (i.e., thememory 1300 and/or the storage 1600) such as a random access memory(RAM) memory, a flash memory, a read only memory (ROM) memory, anerasable programmable read only memory (EPROM) memory, an electricallyerasable programmable read only memory (EEPROM) memory, a register, ahard disk, a removable disk, or a compact disc-read only memory(CD-ROM). An illustrative storing medium may be coupled to the processor1100, and the processor 1100 may read information from the storingmedium and write the information into the storing medium. Alternatively,the storing medium may also be integral with the processor 1100. Theprocessor and the storing medium may also be resided within anapplication specific integrated circuit (ASIC). The ASIC may also beresided within a user terminal. Alternatively, the processor and thestoring medium may also be resided within the user terminal as anindividual component.

In the apparatus and the method to determine drowsiness of a driver asdescribed above, the configuration and the method of the above-mentionedexemplary embodiments are not restrictively applied. In other words, allor some of the respective exemplary embodiments may be selectivelycombined with each other so that they may be variously modified. Asdescribed above, according to the exemplary embodiments of the presentdisclosure, the drowsiness of the driver which may occur duringoperation of the vehicle may be determined within a reduced timeduration as compared with current drowsiness determination methods.Further, the misrecognition of the drowsiness that may occur when basedon the cluster or audio video navigation (AVN) operation may be reduced.Additionally, effects obtained by the present disclosure are not limitedto the above-mentioned effects. In particular, other effects that arenot mentioned may be obviously understood by those skilled in the art towhich the present disclosure pertains from the following description.

Hereinabove, although the present disclosure has been described withreference to exemplary embodiments and the accompanying drawings, thepresent disclosure is not limited thereto, but may be variously modifiedand altered by those skilled in the art to which the present disclosurepertains without departing from the spirit and scope of the presentdisclosure claimed in the following claims.

What is claimed is:
 1. An apparatus for determining drowsiness of adriver, comprising: an imaging device configured to obtain an image ofthe driver; and a controller configured to determine from the image ofthe driver when the driver is in a fatigue state, when the driver is inan eyelid closure state, and when the driver is in a wake-up state,wherein the controller is configured to determine that the driver is ina drowsiness state when, after the fatigue state of the driver continuesover a predetermined first time interval, and the eyelid closure stateand the wake-up state of the driver occur alternately and repetitivelywithin a predetermined second time interval.
 2. The apparatus accordingto claim 1, wherein the controller is configured to determine when thedriver is in the fatigue state based on at least one of when apercentage of an eye closure of the driver exceeds a preset percentageduring a predetermined third time interval and when a face motion of thedriver is within a predetermined fourth time interval.
 3. The apparatusaccording to claim 2, further comprising: a vehicle driving stateobtainer configured to determine a driving state of a vehicle operatedby the driver, wherein the controller is configured to determine whenthe driver is in the fatigue state based on at least one of when thedeviates from a driving lane or when the vehicle is driven in a zigzagdirection from the driving state of the vehicle.
 4. The apparatusaccording to claim 1, wherein the controller is configured to determinewhen the driver is in the wake-up state based on at least one of an eyeblink of the driver that occurs within a predetermined fifth timeinterval and when a face motion of the driver exceeds a preset degree.5. A method for determining drowsiness of a driver, comprising:obtaining, by an image of the driver; determining, by a controller, fromthe image when the driver is in a fatigue state, when the driver is inan eyelid closure state, and when the driver is in a wake-up state; anddetermining, by the controller, that the driver is in a drowsiness statewhen after the fatigue state of the driver continues over apredetermined first time interval, the eyelid closure state and thewake-up state of the driver occur alternately and repetitively within apredetermined second time interval.
 6. The method according to claim 5,wherein when the driver is in the fatigue state is determined based onat least one of when an eyelid closure time of the driver exceeds apredetermined third time interval and when a face motion of the driveris within a predetermined fourth time interval.
 7. The method accordingto claim 6, further comprising: determining, by the controller a drivingstate of a vehicle which is driven by the driver, wherein when thedriver is in the fatigue state is determined based on at least one ofthe vehicle deviates from a driving lane or the vehicle is driven in azigzag direction from the driving state of the vehicle.
 8. The methodaccording to claim 5, wherein when the driver is in the wake-up state isdetermined based on at least one of when an eye blink of the driveroccurs within a predetermined fifth time interval or when a face motionof the driver exceeds a preset degree.